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ABST'iACT 

Data froa a total of 6 #726 evaluation forss collected 
onei: three teras supplied inforaation concerning instructors and 
courses. Iteas on the evaluation forvs vere separated into five 
factors. The variance^covariance aatrix of these factors for eiMCth . 
clfiss vas compared vith the coubined variance^covariance satrix of 
thd other classes taught by the instructor. It was also compared vith 
tbe coBbined variance^covariance Batrix of other sections of the sa«e 
course taught by different instructors. In aost cases, the 
vfiriance-covariance uatrix for a given class resembled the coibined 
v&riance-covariance uatrix of other classes taught by the instructor. 
(Author) 
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While there does not seem to exist a clear-cut definition 
of good teaching, student evaluations are used to measure some 
aspect of the instructor's effectiveness. Some institutions, 
or departments in institutions, use student evaluation in 
promotion and tenure deliberations. While any evaluation of 
an instructor is an implied comparison with other instructors 
that the evaluator has experienced, a direct quantitative 
comparison among instructors would seem to be valid only if 
the evaluators are from the same population. 

It is a distinct possibility, however, that the students 
who are evaluating instructors come from different populations. 
The Department of Mathematical Sciences at Ball State University 
serves mathematics and computer science majors and minors, 
elementary education majors, busi.».t^ss majors, and students who 
need to take mathematics to fulfill a general studies requirement. 
It seems possible that a mathematics major would rate the 
effectiveness of an instructor teaching a matheiuatics course 
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differently than an elementary education major would rate the 
effectiveness of the instructor in a mathematics course for 
elementary education majors* 

Instructors also complain that they have had a "bad" ciass* 
There is evidence to support the claim that an instiructor using ' 
the same methods may get different ratings in different sections 
of the same course (Center for Improved Learning, 1972)* 

To test for a consistent population, one can test for 
homogeneity of variance* If homogeneity of variance cannot be 
established and the sample sizes are different, a comparison 
of the means of one sample with another may not be valid* 

Theoretical Framework 

The researchers assisted in the development of an evaluation 
form where five components were identified (Nelson, 1974) * These 
five components were named Instructor Presentation, Interaction- 
Evaluation, Classroom Details, Student Motivation, and Course 
Information* A copy of the foirm used in the evaluation appears 
in Appendix A* The statements on the evaluation form were re- 
lated to the components as follows: 



Component 



Statements 



Instructor Presentation 



1-10 



Interaction-Evaluation 



11-18 



Classroom Details 



19-24 



Student Motivation 



25-27 



Course Information 



28-32 
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Scoring of the evaluation form was done using the method 
of summated ratings with normal deviate weights (Nelson, 1974) • 
The use of normal deviate weights insures a multi-normal dis- 
tribution# 

Morrison(1967) suggests a generalization of Dartlett's 
test for homogeneity of variance with p variates* This procedure 



involves the calculation of M, where M = ^{^^i^ l^i|» 
n^ is the number of observations in each of k samples, (s| is the 
determinant of the covariance matrix when the samples are 
pooled, and Js^j is the determinant of the ith sample covariance 
matrix* Box(1949) has shown that the quantity MC""^ is approximately 
distributed as a chi-squared variate with degrees of freedom 
i (k - 1, p ,p . 1,, Where c"! = 1 - '^"'e^^Vl) '^^ ' ^' 'Ji ' 

This approximation appears to be good if k and p do not each exceed 

5, For other values of k and p, he suggests the quantity M 

b 



approaches the F distribution with f, and f, degrees of freedom, 

fl 1 f 1 + 2 
where b = ± ^ , f , = ^ (k - 1 ) p (p + 1), f = , 



1 - - 11 ^2 - Ai 

^2 



A, = ^ 
^ 6 (k 



:-l)(p+l) '*'"i ^^i 2 6(k-x) $^2 (j^^^, 



Using the statistical procedure described above, the 
hypothesis 

where ^ ^ represents the covariance matrix of the ith sample, 
of equal covariance matrices can be tested against the hypothesis 

that at least two of tho covariance matrices are different. 

4 
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Procedure 

The evaluation for© was administered three times. Winter 
quarter 1972-73, Fall quarter 1973, and Spring quarter 1974, 
in the Department of Mathematical Sciences at Bali State University* 
The number of usable forms totaled 6726; 2445 Winter quarter 
1972-73, 2486 Fall quarter 1973, and 1795 Spring quarter 1974# 
Noxrroal deviate weights were computed for each quarter* Since 
these weights did not vary by more than •08^ the Fall quarter 
1973 weights were used in this study* All personnel teaching 
classes in the Department of Mathematical Sciences, from graduate 
assistant to full professor weren required to administer these 
forms in their classes* The data selected for this report 
included undergraduate courses thr.t had the most sections* 

The courses selected were as follows: 

Elementary education 

Mathematics content: 201, 202, 203, 204, 209 
Mathematics methods: 391 

Mathematics majors and minors : 

Analytic geometry: 113 
Calculus: 170, 171, 172 
Structures: 211 

Gefaeral Studies Mathematics : 100, 101 

Business Mathematics : 131, 132 

Table 1 lists the instructors, by code number, and the courses 
that were taught by these instructors* 

Students did not know wrich instructors would be teaching 
a particular class, since instructors' were assigned to classes 
after the students had completed registration* Students rarely 
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Table 1 



Courses 


and Number of Sections taught bv Instructors 




Tns'fcriiC'fcor 


201 


202 


203 


204 


Course 
209 391 113 


170 


171 


172 211 320^100 


1 ni 1 1 

X \/X X J X 


14 


1 


1 


1 








1 


1 




17 


2 


1 




1 


1 2 










24 


1 




5 


2 


2 5 










28 




2 


1 




1 










• 31 








2 


3 4 










32 


1 


1 


7 


1 


5 










36 










1 


1 




1 




37 




2 














1 


45 




3 






1 


1 


1 


1 




50 








5 


5 










53 










1 


1 


1 


1 




55 


1 


2 






1 


1 




1 


1 


63 




3 


2 




1 


2 




1 




68 


2 






1 


2 


1 




2 


2 


71 




1 














1 


73 




1 


2 










1 




78 








2 












81 




1 




2 


6 6 










85 


3 


1 








1 


1 






88 


1 


2 




6 


2 










93 


2 


1 












1 




95 










2 










97** 


1 








1 











* Course 320 included to give complete data for instjructor 73. 
**Data incomplete for instructor 97 
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were allowed to change sections of a course after the quarter 
had begun. Although the assignments of students to classes 
may not have been random, students did not choose a class for 
the instructor. Most classes had between 15 and 30 students. 
The only large classes were two sections of course 131 which 
had 60 and 70 students, and two sect^.ons of course 132 which 
contained 51 and 58 students* 

Covariance matrices were computed fox each sectior of 
every course listed above, and then were pooled to give the 
covariance matrix for each course and for each instructor. 
The following hypotheses were then tested using the F dis- 
tribution, and, where appropriate, the chi*-squared distribution: 

: There are no differences in the covariance matrices 
of che five components of the evalu^ition form among 
different sections of the same course taught by 
different instructors. 



H2 : There are no differences in covariance matrices of 
the five components of the evaluation form among 
different classes taught by the same instructor. 

Results 

The results for each course are presented in Table 2. 
Hypothesis H^^ is rejected in every case, and, in all but one 
case at the .01 level of significance. It is interesting to 
note that the chi-squared approximation agreed with the F 
approximation in every case. 

The results for each instructor are presented in Table 3. 
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Table 2 

Test for Differences Between Ingtrugtors 



Course 


Number of 
Sections 


Total Numbe^' 
of students 




P 


df 


OAT 


15 


390 




2.618** 


210, 82 796 




^2 


590 




2.723** 


315,107294 


O A^ 


18 


437 




2.209** 


255,53819 


204 


22 


516 




2.505** 


315.101593 
* 


O A A 

209 


18 


398 




2.474** 


255, 76144 


391 


25 


513 




2.702** 


360,94318 


113 


9 


203 




2.899** 


120,28614 


170 


8 


147 




2.568** 


105,20012 


171 


4 


94 


117.61**45 


2.i/85** 


45,4660 


172 


4 


73 


95.39**45 


2.178** 


45,6276 


211 


2 


29 


41.56**15 


2.755** 


15,3352 


100 


3. 


66 


47,41* 30 


1.576* 


30,8676 


101 


3 


33 


63.34**30 


2.086** 


30,2486 


131 


2 


130 


39.37**15 


2.624** 


15,64836 


132 


3 


129 


60.64**30 


2.017** 


30,13672 



" Significant at .05 level 
Significa.it at .01 level 
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Table 3 





Test 


for Differences 


Between Clrisses 






Instructor 


Number of 
Classes 


Total Number 
of students 




P 


df 


14 


5 


154 


138.11**60 


2.181** 


60,1245 


17 


7 


181 




1.882** 


90,88038 • 


24 


15 


371 




3.339** 


210,79502 


28 


4 


101 


107.07**45 


2.372** 


45,15789 


31 


9 


177 




1.742** 


120,32696 


32 


15 


319 




1.682** 


210,36649 


36 


4 


76 


108.53**45 


2.342** 


45,1771 


37 


3 


54 


45.55* 30 


1.695* 


30,721 


45 


7 


154 




1.670** 


90,19644 


50 


10 


225 




2.078** 


135,'42150 


53 


4 


77 


88.16**45 


1.S47** 


45,8321 


55 


7 


217 




1.953** 


90,15882 


63 


9 


190 




2.438** 


120,30152 


68 


10 


205 




1.464* 


135,26139 


71 


2 


92 


20.78 15 


1.384 


15,17626 


73 


6 


192 




2.201** 


75,16996 


78 


2 


51 


15.54 15 


1.122 


15,9664 


81 


15 


359 




2.679** 


210,75199 


85 


6 


110 


113.81**75 


1.505* 


75,10069 


93 


4 


97 


105.44**45 


2.337** 


45,17743 


95 


2 


46 


24.41 15 


1.621 


15,4406 



* significant at .05 level 
O ** Significant at .01 level 
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In the 22 cases where data was available, the hypothesis 
H2 was rejected 19 times at the •OS level of significance, and 
16 of these times at the .01 level of significance* In only 
one case did the chi-SG[uared approximation differ from the F 
approximation; this case was therefore considered significant 
only at the •OS level* In three cases, the hypothesis H2 
could not be rejected* 

Conclusions 

It is interesting to note that hypothesis H^^, the assumption 
of equal covariance matrices for each section of a course, could 
be rejected at least at the •OS level of significance in each 
case* At the same time, the assumption of equal covariance 
matrices for each class taught by an instiructor could l:e rejected 
in 19 out of 22 cases* In the cases where H2 could not be re-- 
j acted, data could only be reported for two classes for the 
instructor. In cases 78 and 9S, the instructors had taught two 
sections of the same class, while in case 71, the instructor had 
taught two different courses* 

In cases 24, 31, 32, SO, and 81, the instructors are 
restricted to the elementary education majors* Even so, the 
results reported for these instructors indicate unequal covariance 
matrices* These instructors are the only personnel of the 
department with an elementary teaching licence and thus qualified 
to teach the elementary methods course, math 391. However, the 
results Indicate unequal covariance matrices for this course* 
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At least in this study, the classes seemed to make the 
difference in the covariance matrices* Students generally 
seemed to give favorable ratings to an instructor; the average 
rating across items for all 6726 students was 1.01, which 
coincides with the "Agree* category* It is possible that the 
differences in the covariance matrices between classed were 
caused by a small percentage of students that rate the 
instructor in the "Disagree" or "Strongly Disagree" category 
on several statements* 

There is also the possibility that the evaluation form 
used was not suited to comparing differences among classes* 
The researchers would like to see a similar study done using 
another evaluation form* 

Student evaluation of instruction is only one measure of 
instructor effectiveness* The results presented indicate that 
the means for classes may be difficult to compare* If the 
results of a student evaluation are to be used for comparison 
of personnel, the results should be substantiated by another 
means of evaluating the instructor such as class visitation* 
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INSTRUCTIONS 



Please follow the instructions below carefully. This will help us 
in processing these forms. It is requested that you do NOT sign your najtne. 



1. Be sure that you are using a No. 2 pencil. 



2. On the Evaluation Form, record the course number, section number, and 
instructor code number for the class in which you are enrolled. 



3* On the Evaluation Form, circle the number corresponding to your reason 
for taking the course. 



U. In the upper right hand comer of the machine form you will find a section 
titled, " Identification Number." In the top three squares, record the 
course number. Now darken one number in each of the top three rows 
corresponding to the digits that you recorded for the course number. 



Do the sajne for the section number, instructor code number and the nuzriber 
for your reason for taking the class. Use two digits to record your 
se-^tion number. For exariiple, if you are in section 2, record this as 02. 



6. Circle your response to each statement on the Evaluation Form, then Pecord 
your response in the LEFT column of the machine form. The nuiiiber in 
parentheses beside each statement in the Evaluation Form shows ^^ere you 
are to record the answer on the machine form. For exairple, statement 2 
on the Evaluation Fom is recorded as 5 in the first column of the 
machine form. 



7. Wien you have responded to all items and have made any comments that you 
wish to make, place your Evaluation Form face down in the appropriate pile 
in the front of the room and the corresponding machine form face down in 
the other pile. 
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MTHEMATICAL SCIENCES INSTRUCTOR EVALU/lTiaN FORM 



Classification: 
Course number: 
Section nuiriber: 
Instructor Code nuiriber: 
}fy reason for taking this course is that 

0. it is a requirement on a major or minor in a mathematical science. 

1. it is a requirement on a major or minor other than one in a mathematical 
science. 

2. it is an elective on a major or minor in a mathematical science. 

3. it is an elective other than a major or minor in a mathematical science. 
U. a reason other than above. 

Circle one of the following categories for each statement below, then darken the 
appropriate square on the machiie form. The code is: 



a. 


Strongly Agree 


SA 


b. 


Agree 


A 


c . 


Neutral 


N 


d. 


Disagree 


D 


e. 


Strongly Disagree 


SD 



Instructor Presentation 

(1) 1. The course content was presented in a SA A N D SD 

coherent manner, with major points 
emphasized, and relationships made clear. 

{$) 2. The instructor seemed well -prepared SA A N D SD 

for the lectures, discussions, or other 
activities. 

(9) 3. The instructor provided enough examples SA A N D SD 
and illustrations to clarify the 
material for me. 
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(13) The iiistmctor seemed enthusiastic SA. A N D SD 

about teaching. 

(17) 5* The instructor seeined to have a SA A N D SD 

thorough knowledge of the subject. 

(21) 6. The L-structor had a sufficient SA A N D SD 

vocabulary, us3d appropriate 
language, correct gramraar, and 
expressed h5.s ideas clearly* 

(25) ?• The instructor spent a minimum amount SA A N D SD 

of class time on information unrelated 
to the course content. 

(29) 8. The instructor's presentations made SA A N D SD 

class attendance worthwhile. 

{yj) 9. The instructor's pacing of the course SA A N D SD 

material was satisfactory. 

(37) 10. The itethod of instruction was adequate? SA A N D SD 
I see no need for an appreciable change 
in the instructor -s method. 

(Ill) 11. The instructor gave a sufficient number SA A N D SD 
of appropriate assignments basea on or 
related to the material covered. 

(U^) 12. Tests or other evaluative measures were SA A N D SD 
appropriate to the course. 

(U9) 13. The insti-uctor encouraged questions in SA A N D SD 
class. 

(53) lU* The instructor's answers to questions SA A N D SD 

were usually adequate. 

(57) 15. The instructor tried to establish SA A N D SD 

conditions conducive to learning. 

(61) 16. The instructor maintained office hours SA A N D SD 
and/or provided adequate opportunity 
for individual help. 

(65) 17. The instructor had sufficient evidence, SA A N D SD 
in terms of written work, projects, or 
tests, to evaluate w performance. 

(69) 18. The instructor's grading system seemed SA A N D SD 
fair to me. 
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(73) 19* I was able to read what was written SA A N D SD 

on the chalkboard* 

(77) 20. I was able to hear the instructor SA A N D SD 

without difficulty. 

(81) 21. The instructor dressed appropriately SA A N D SD 

for class. 

(85) 22. The instructor was well-groomed. SA A N D SD 

(89) 23. The instructor had a pleasant speaking SA A N D SD 
voice . 

(93) 2U. The instructor did not have distracting SA A N D SD 
mannerisms. 

Motivation 

(97) 25. I made an honest effort to learn in SA A N D SD 

this course. 

(101) 26. I felt intellectually challenged by SA A N D SD 

this course. 

(10^) 27* I was motivated to spend time, or at SA A N D SD 

least tiy to spend tirne, preparing 
for the course. 

Course information 

(109) 28. I felt that the textbook and/or the SA A N D SD 

other materials were adequate for the 
course. 

(113) 29. I felt that the description of the SA A N D SD 

course in the catalog was adequate. 
(Please mark N if you have not read 
the course description in the catalog.) 

(117) 30. I learned as much as I expected to SA A N D SD 

learn from the course. 

(121) 31* The prerequisites for the coui'se were SA A N D SD 

adequate for me. 

(12^) 32. I felt the content of the course, ^ SA A N D SD 

exclusive of instruction, was 
worthwMle to me. 



Er|c 17 



(129) 33. Year at Ball State Universi+y 

A. Freshman 

B. Sophomore 

C. Junior 

D. Senior 

E. Graduate 

(133) 31;. W overall grade average at Ball State University is about 

A. 1.0 

B. 2.0 

C. 3.0 

D. U.O 

E. Viy first quarter at Ball State University 
(137) 35. The number of class meetings thai I have missed was 

A. 0-2 

B. 3-5 

C. 6 - 8 

D. 9 - 11 

E. 12 or more 
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Comments : 

Written comments have been found to be particularly helpful to the instructor. 
Please use this sheet to comment about the instructor, the course, or this 
evaluation form. Your cooperation is appreciated. 
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